CS 269 : Machine Learning Theory

The idea behind boosting is to construct an accurate hypothesis from a set of hypotheses that are each guaranteed to perform slightly better than random guessing on particular distributions of data. This idea originally arose from attempts to prove the robustness of the PAC model. By robustness, we mean the notion that slight alterations to the model definitions should not result in dramatically different results, such as the ability to learn different classes of functions. The question of whether weak learning implies strong learning was first posed in the late 80s by Kearns and Valiant. The first boosting algorithm was developed by Rob Schapire in order to answer this question. This paved the way for the immensely popular AdaBoost algorithm, which was developed by Freund and Schapire a couple of years later. To formalize this discussion we will first review the basic definition of PAC learnability, which we now refer to as strong PAC learnability to distinguish it from the weak version. Note that this is the standard definition that we have discussed in class previously.